Machine Vision based Vehicular Cabin Activity Analysis

Vision for intelligent vehicles is a growing area of research [6] for many practical reasons including the relatively inexpensive nature of camera sensing units and even more the non-contact and non-intrusive manner of observation. The latter is of critical importance when observing occupants inside the vehicle cockpit because no sensing unit should impede the primary task of driving. Conventionally, observation of occupants inside the vehicle has been limited to drivers of the vehicle since the state of driver directly affects the task of driving [1, 3]. However, the holistic state of the driver is better represented by observing the entire situation inside the cabin, including the passengers. As illustrated in Figure 1, passengers and drivers jointly exhibit a variety of behaviors which affects the state of driver and therefore the quality of driving. In the top image, the passenger is verbally communicating information to the driver which translates to a certain percentage of the driver’s mental workload. In the second image, the passenger and driver are jointly observing the contents of a cell phone causing the driver to be visually and mentally distracted. To this end, vision based activity analysis of vehicular cabin occupants is of particular interest for at least two reasons: first it has the potential to save lives and second it posses interesting and challenging research problems from the computer vision and machine learning perspective. In this study, we lay the ground work for analyzing activities inside the vehicle cabin by describing the steps for proper data collection, in defining activities of interest, on extracting reliable vision based features and for modeling joint activities.